litewitness: add Prometheus metrics. - #71
Conversation
|
@warpfork thanks for adding this! I just tested it for one of my test witnesses and it works: Corresponding issue: #5 @FiloSottile could you have a look and see if this can get merged? |
|
@FiloSottile what do you think, could we get this merged soon? It would be very useful for us (Glasklar)! From the perspective of Glasklar's witness deployments, for us it would also have been helpful to be able to serve metrics at localhost even if the witness endpoint that logs use is publicly reachable. @warpfork what do you think about adding an option to allow that? I guess it could be an option in addition to the existing |
|
I'll take this for a spin tomorrow on my witness -- and think about what metrics I like / dislike / miss. What's the best way to add more metrics, wait until this PR is merged and then file my own PR? |
|
Sorry for the delay, I've reviewed + tested running this on rgdd.se/poc-witness. I've also hooked up the metrics to grafana. TL;DR: diff looks good (just one minor fix suggestion -- see below), and the added metrics cover the most essential things we need. So, no suggestions wrt. adding/removing/modifying any metrics for now. Main thing we need is what @eliasrudberg already mentioned above, i.e., to be able to serve metrics on a separate addr:port endpoint. I attached a patch for this below, and I'm running this PR with this patched applied. Patch suggestion: 5676ac2 CC @FiloSottile. |
Patterns heavily riffed from the sunlight codebase. Standard go and process stats are collected, and custom stats: these include the number of logs monitored, how large they are, how long requests are taking to service, etc. The prefix levels on the metrics might be a little high; what I'm following here is the sunlight project's distinction between the command's whole metrics, and the witness component. Right now, the only metrics I've introduced to litewitness are on the witness component (and that sounds doubly redundant due to the project name), but that might not remain the case forever. I disable the exposure of the metrics endpoint if the obscurity mode flag is engaged. Some metrics do contain log names, and so it matches the same enumeration concern that obscurity mode describes as its purpose. (Possibly future work: I'm not super satisfied with how sometimes the logging labels and the prometheus labels have parallel evolution, for example in `processAddCheckpointRequest`. But attempting to unify those feels like a potential portal to frameworkitis, so leaving that as a future itch to scratch. Or, perhaps not.) Closes FiloSottile#5
|
I added |
4 participants
Metrics! Whee!
Patterns heavily riffed from the sunlight codebase.
Standard go and process stats are collected, and custom stats: these include the number of logs monitored, how large they are, how long requests are taking to service, etc.
The prefix levels on the metrics might be a little high; what I'm following here is the sunlight project's distinction between the command's whole metrics, and the witness component. Right now, the only metrics I've introduced to litewitness are on the witness component (and that sounds doubly redundant due to the project name), but that might not remain the case forever.
I disable the exposure of the metrics endpoint if the obscurity mode flag is engaged. Some metrics do contain log names, and so it matches the same enumeration concern that obscurity mode describes as its purpose.
(Possibly future work: I'm not super satisfied with how sometimes the logging labels and the prometheus labels have parallel evolution, for example in
processAddCheckpointRequest. But attempting to unify those feels like a potential portal to frameworkitis, so leaving that as a future itch to scratch. Or, perhaps not.)